Systems and Methods to Increase Discoverability in User Interfaces

ABSTRACT

The present disclosure provides systems and methods to improve discoverability of selectable user interface elements. In particular, the present disclosure provides computing devices that, in some implementations, intelligently identify user interface elements that are selectable (e.g., based on intelligent understanding of the user interface content) and visually modify one or more of such selectable user interface elements to indicate to the user that they are selectable. The visual modification can highlight or otherwise draw attention to the user interface element(s), thereby improving their discoverability.

FIELD

The present disclosure relates generally to user interfaces. Moreparticularly, the present disclosure relates to intelligentidentification and highlighting of user-selectable user interfaceelements based on, for example, location of a user object relative to adisplay.

BACKGROUND

Computing devices (e.g., smartphones) can provide (e.g., display on adisplay screen) a user interface that enables the computing device tointeract with a user, including, for example, receipt of commands fromthe user, providing (e.g., displaying) information to the user, and/orother functions.

At any given point in time, a user interface can include a number ofdifferent user interface elements. As examples, user interface elementscan include icons, buttons, textual elements, items of content, pictures(e.g., pictures being captured in real-time via a camera and/orpreviously captured pictures), banners, titles, headers, entry fields,windows, menus, controls, graphics, and/or the like.

In some instances, some or all of the user interface elements in a userinterface can be selectable by the user. As one example, a particularuser interface element can correspond to an entity (e.g., location,point of interest, person, object, etc.) and the user can select theparticular user interface element to receive additional informationabout the entity. In other examples, various other actions can betriggered by or result from selection of a user interface element by theuser. As a further example, in some instances, a user can select a userinterface element that is displayed on a touch-sensitive display screenby touching or tapping the display screen at the location at which theuser interface element is displayed.

However, in some instances, it may not be readily apparent to the userwhich of the user interface elements are selectable. As one example,user interface elements may be presented uniformly regardless of whetheror not they are selectable. As another example, a certain type of userinterface element may traditionally not be selectable and, absent someindication to the user to the user that a particular element of thattype is selectable, the user will assume that such particular element isnot selectable. Thus, a user may, in fact, desire to trigger aparticular action that would result from selection of a user interfaceelement but not realize that the user interface element is selectable.

In one particular example, a user interface provided by a computingdevice can include a picture of a food dish and selection of the pictureby the user (e.g., via touch input) would cause further actions to beperformed by the computing device (e.g., retrieval and storage of arecipe for cooking the food dish so that the user can later view therecipe). The user may, in fact, be interested in cooking the dish at alater time and would therefore appreciate the opportunity to have therecipe received and stored. However, if the user does not realize thatsuch action is available via selection of the user interface element,then the opportunity to satisfy the user's desire will go unrealized.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to a computingdevice. The computing device includes one or more processors, a display,and one or more non-transitory computer-readable media that collectivelystore instructions that, when executed by the one or more processors,cause the computing device to perform operations. The operations includeproviding a user interface that comprises a plurality of user interfaceelements for presentation on the display. The operations includeidentifying one or more of the user interface elements that areselectable by a user of the computing device. The operations includevisually modifying at least a first user interface element of the one ormore selectable user interface elements to have a gleam appearance inwhich the first user interface element appears to reflect light.

Another example aspect of the present disclosure is directed to acomputer-implemented method. The method includes providing, by one ormore computing devices, a user interface that comprises a plurality ofuser interface elements for presentation on a display. The methodincludes identifying, by the one or more computing devices, one or moreof the user interface elements that are selectable by a user. The methodincludes determining, by the one or more computing devices, an objectlocation of a user object relative to the display. The user object is afinger or a stylus located adjacent to the display but not physicallytouching the display. The method includes determining, by the one ormore computing devices, whether the object location of the user objectcorresponds to one of the one or more user interface elements that areselectable by the user. The method includes, when the object locationcorresponds to a first user interface element of the one or moreselectable user interface elements, visually modifying, by the one ormore computing devices, the first user interface element to visuallyhighlight the first user interface element.

Another example aspect of the present disclosure is directed to anapparatus. The apparatus is configured to provide a user interface thatcomprises a plurality of user interface elements for presentation on adisplay. The apparatus is configured to identify one or more of the userinterface elements that are selectable by a user. The apparatus isconfigured to modify display of selectable user interface elements as afunction of a location of a finger or stylus located adjacent to thedisplay but not physically touching the display.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, user interfaces,and electronic devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example computing system accordingto example embodiments of the present disclosure.

FIG. 2 depicts a flow chart diagram of an example method to increasediscoverability of selectable user interface elements according toexample embodiments of the present disclosure.

FIGS. 3A and 3B depict an example modification of an example userinterface element based on an orientation of an example computing deviceaccording to example embodiments of the present disclosure.

FIGS. 4A and 4B depict an example modification of an example userinterface element based on an orientation of an example computing devicerelative to an example user object according to example embodiments ofthe present disclosure.

FIG. 5 depicts a flow chart diagram of an example method to increasediscoverability of selectable user interface elements according toexample embodiments of the present disclosure.

FIGS. 6A-D depict an example modification of example user interfaceelements based on a location of an example user object relative to theuser interface elements according to example embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Example aspects of the present disclosure are directed to systems andmethods to improve discoverability of selectable user interfaceelements. In particular, the present disclosure provides computingdevices that, in some implementations, intelligently identify userinterface elements that are selectable (e.g., based on intelligentunderstanding of the user interface content) and visually modify one ormore of such selectable user interface elements to indicate to the userthat they are selectable. The visual modification can highlight orotherwise draw attention to the user interface element(s), therebyimproving their discoverability. Moreover, this can be achieved withouthighlighting or otherwise drawing attention to the user interfaceelement(s) all of the time, so that the readability etc. of the userinterface element(s) is not negatively affected other than when thehighlighting etc. is provided. For instance, in some implementations,user interface element(s) can be provided without any highlighting whenthe computing device is in a normal reading position, whereby the usercan consume the content displayed on the display without theirexperience being affected by highlighting, and the computing device canthen provide the user interface element(s) with highlighting when thecomputing device is tilted sideways away from the normal readingposition, whereby the user can identify the selectable user interfaceelement(s) at a time when they are less concentrating on consuming thecontent displayed on the display.

Furthermore, in some implementations, a computing device can visuallymodify a user interface element based on or as a function of a locationof a user object (e.g., finger, stylus, face, etc.) relative to adisplay of the device and/or an orientation of the device (e.g.,orientation relative to the ground and/or orientation relative to a userobject). One example visual modification of a user interface element caninclude visually modifying the user interface element to have a gleamappearance in which the user interface element appears to reflect light.For example, a reflection location at which light appears to bereflected can be updated in real time based on or as a function of thelocation of the user object relative to the display of the device and/orthe orientation of the device. By changing a reflection location as afunction of the location of the user object relative to the display ofthe device and/or the orientation of the device, the effect of movementcan be provided. This helps to focus the user's attention on the userinterface element because humans notice movement more readily than theynotice relatively static imagery. This allows the user's attention to bedrawn to the user interface element with relatively little adaptation ofthe static content of the user interface element, allowing the user morereadily to view/consume the static content of the user interface elementfor a given amount of attraction of the user's attention to the userinterface element.

Thus, according to aspects of the present disclosure, a computing devicecan intelligently recognize content on the device's screen that isrelated to one or more entities and can inform the user that the userinterface elements that correspond to such content are selectable totrigger additional actions (e.g., searching for, storage of, and/orcommunication of information associated with a corresponding entity). Inone particular example, a text message received by the computing devicefrom a contact of the user can include a picture of a food dish thecontact cooked. An intelligence layer of the computing device can, uponpresentation of the picture within the user interface, automaticallyrecognize the food dish and ascertain that additional information (e.g.,a recipe for the food dish) can be accessed, stored, communicated, etc.The computing device can visually modify the picture of the food dish(e.g., to have a gleam appearance) so that the user discovers theability to select the picture to cause access, storage, or communicationof the information (e.g., recipe). In such fashion, user access to anduse of intelligent content-recognition and information provisioningsystems can be increased, thereby enhancing user satisfaction.Furthermore, since user access to information can be streamlined in suchfashion, the number of redundant requests, queries, searches, or otheractions across multiple different applications can be reduced, therebyproviding savings of computational and network resources.

More particularly, a computing device can provide a user interface thatincludes a plurality of user interface elements for presentation on thedisplay. For example, the user interface can be a user interfacegenerated by an application of the computing device. As examples, userinterface elements can include icons, buttons, textual elements, itemsof content, pictures (e.g., pictures being captured in real-time via acamera and/or previously captured pictures), banners, titles, headers,entry fields, windows, menus, controls, graphics, and/or the like.

As one example user interface, a web browser application can provide auser interface that includes a web page that includes a number of userinterface elements. As another example, a text messaging application canprovide a user interface that includes text messages sent and/orreceived to one or more contacts. As yet another example, a cameraapplication can provide a user interface that depicts a field of view ofa camera of the computing device. Many other and different userinterfaces can be provided as well.

According to another aspect of the present disclosure, the computingdevice can identify one or more of the user interface elements that areselectable by a user of the computing device. For example, in someimplementations, the computing device can include a user interfacemanager. In some implementations, the user interface manager can beseparate from whichever component (e.g., application) generated the userinterface. The user interface manager can continuously or periodicallyanalyze the user interface that is currently or about to be displayed bythe computing device to identify user interface elements that are orshould be selectable by the user of the computing device.

According to an aspect of the present disclosure, in someimplementations, identifying the one or more of the user interfaceelements that are selectable can include employing an artificialintelligence layer to recognize entities within the user interface. Moreparticularly, in some implementations, a computing device can include anartificial intelligence layer that automatically and intelligentlyidentifies user interface elements (e.g., certain items of content)within a user interface that correspond to certain entities. Asexamples, entities can include locations, points of interest, persons,objects, products (e.g., articles of clothing, food items, etc.),brands, programming content (e.g., movies), bands, events, and/or otherpeople, places, or things.

In some implementations, the artificial intelligence layer can performmobile vision techniques to recognize entities in imagery included inthe user interface and/or text recognition techniques to recognizeentities in text included in the user interface. In someimplementations, the artificial intelligence layer can include one ormore machine-learned models. As examples, the machine-learned models caninclude machine-learned image recognition models, machine-learned objectdetection or recognition models, machine-learned facial detection orrecognition models, machine-learned text recognition models, and/orvarious other types of machine-learned models that are useful forrecognizing and understanding entities included in user interfacecontent.

To provide an example of the above-described concepts, a web browserapplication can provide a user interface that includes a web page, suchas, for example, a style magazine web page. The user interface managercan handle and manage presentation of the user interface within adisplay, including analyzing the user interface to identify userinterface elements that are or should be selectable by the user. Inparticular, the artificial intelligence layer can scan the stylemagazine web page to identify any content that corresponds to entities.For example, the artificial intelligence layer can use a machine-learnedobject recognition model to recognize certain articles of clothing(e.g., a particular model of watch manufactured by a particular brand)that are depicted in the style magazine web page. The user interfacemanager can identify user interface elements (e.g., a picture thatincludes the watch) that correspond to the recognized articles ofclothing (e.g., the watch) and can, upon presentation of the stylemagazine web page in the user interface, cause the identified userinterface elements (e.g., the picture of the watch or a portion thereofthat includes the watch) to be selectable by the user. For example, aswill be described further below, selection of the user interface elementthat corresponds to the watch can result in the computing deviceperforming various actions, such as, for example storing informationabout the watch, redirecting the user to a shopping web page thatenables purchase of the watch, pinning the watch to the user's styleboard, and/or various other actions.

In some implementations, identifying user interface elements that areselectable can include generating additional user interface elements forpresentation within the user interface and/or transforming a certainuser interface element into a multiple user interface elements. Tocontinue the example above, once the watch has been recognized in thepicture included in the style magazine web page, the computing devicecan generate an additional user interface element that corresponds tothe portion of the picture that includes the watch, if no such userinterface element existed previously. Thus, new and/or modified userinterface elements can be used to enable the user to select certainspecific entities (e.g., the watch) included in a larger group ofentities (e.g., six different articles of clothing recognized in thepicture included in the style magazine web page).

In some implementations, identifying user interface elements that areselectable can include accessing attributes or parameters associatedwith the user interface elements. For example, in some instances, thecomponent (e.g., application) that generated the user interface canprovide attribute or parameter data for each user interface elementincluded in the user interface, including, for example, data thatindicates whether each user interface element is selectable. Thus, theuser interface manager can access or read this data to identify which ofthe user interface elements are selectable.

According to another aspect of the present disclosure, the computingdevice can visually modify at least a first user interface element ofthe one or more selectable user interface elements. As an example, insome implementations, all of the user interface elements that areselectable by the user can be visually modified. In otherimplementations, only a subset of the selectable user interface elementscan be visually modified (e.g., based on various contextual data asdescribed below).

In some implementations, the computing device can visually modify theone or more user interface elements as soon as elements load within theuser interface (e.g., as soon as the application, web page, etc. beingdisplayed loads). In other implementations, the computing device canwait to visually modify the one or more user interface elements untilsome other event occurs such as, for example, when a user invokes thekeyboard, when the user performs a scrolling action, when the userreaches an end of the current interface or page, and/or other actions orevents.

According to an aspect of the present disclosure, one example visualmodification that the computing device can perform is to visually modifya user interface element to have a gleam appearance in which the userinterface element appears to reflect light. For example, the gleamappearance can visually mimic the appearance of sun reflecting off of awatch face, a body of water, or other reflective item. Visuallymodifying a user interface element to have a gleam appearance can bringthe user's attention to the user interface element and the fact that itis selectable.

Another example visual modification includes outlining the userinterface element. Another example visual modification includes changingthe color of the user interface element. Another example visualmodification includes highlighting the user interface element bybrightening the user interface element and/or an area around the userinterface element. For example, a flashing of a highlighting color,shading, and/or texture on and/or around the user interface element canindicate that the user interface element is selectable.

According to another aspect of the present disclosure, in someimplementations, the visual modification can be based on an orientationof the device. For example, the visual modification can change inresponse to a change in the orientation of the device.

As one example, the orientation of the device can be an orientationrelative to a direction of gravity (e.g., the direction towards theground). For example, in some implementations, the computing device caninclude a gyroscope. The gyroscope can provide data that indicates theorientation of the device relative to the direction of gravity. Thegyroscope can include one or more moving (e.g., spinning) components orcan include only static components. As one example, the gyroscope caninclude an arrangement of accelerometers (e.g., three or moreaccelerometers arranged to detect acceleration in three dimensions).

As another example, the orientation of the device can be an orientationrelative to a user object. The user object can be positioned in a numberof different positions relative to the device. As one example, in someinstances, the user object can hover over the display. That is, the userobject can be placed adjacent to the display but not physically touchingthe display.

For example, the user object can be a finger or stylus associated withthe user. For example, the finger or stylus can be located at a numberof different locations relative to a display of the computing device,including, for example, hovering over the display as described above. Insome implementations, the computing device can include a RADAR sensingsystem. The computing device can use the RADAR sensing system todetermine an object location of the user object (e.g., finger orstylus). The computing device can determine the orientation of thecomputing device relative to the user object based on the determinedobject location. The visual modification of the user interface elementcan be modified (e.g., in real time) as such orientation changes.

In another example, the user object can be a face of the user. In someimplementations, the computing device can include one or more cameras.The computing device can determine a face location of the user's facebased on imagery captured by the one or more cameras. The computingdevice can determine the orientation of the computing device relative tothe user's face based on the determined face location. The visualmodification of the user interface element can be modified (e.g., inreal time) as such orientation changes.

As one example, visually modifying a user interface element based on theorientation of the device can include, for a user interface element witha gleam appearance, changing a reflection location at which the userinterface element appears to reflect light based at least in part on theorientation of the display relative to the object location of the userobject. Thus, in one example, as the orientation of the device changes(e.g., due to the user tilting the device and/or moving her fingeraround the display) the reflection location can be updated, furthergiving the appearance that the user interface element is gleaming withlight.

As another example, visually modifying the user interface element basedon the orientation can include visually modifying the user interfaceelement at certain orientations but not at other orientations. Thus, inone example, if the user tilts the device to the right or left, the userinterface object can be visually modified, while if the user holds thedevice in a neutral orientation, the user interface object is notvisually modified.

In some implementations, the visual modification can only be temporaryin nature. In one example, as soon as a user interface (e.g., web page)loads, the computing device can briefly (e.g., for one half second)visually modify the selectable user interface elements (e.g., recognizedproducts) and then return the elements to a neural or normal state. Thiscan provide a quick indication to the user of which elements areselectable without being overly disruptive of the user experience.

In some implementations, the particular type of visual modification usedfor a user interface element can be based on or a function of a contenttype associated with such user interface element. For example, textualelements can be modified in a first way while pictorial or graphicalelements can be modified in a second, different way.

According to another aspect of the present disclosure, in someimplementations, the visual modification can be based on a location ofuser object to the display of the computing device. As one example, thecomputing device can visually modify a selectable user interface elementwhen the user object is hovering over the user interface element. Thus,for example, a user may use her finger to hover over the display of thedevice. When the user's finger is located over (or within a certaindistance of) a user interface element that is selectable, the computingdevice can visually modify such user interface element (e.g., using anycombination of one or more of the visual modifications describedherein).

Thus, a user may hover her finger over the display screen and move herfinger to different locations to discover (e.g., via the resultingvisual modification of the user interface elements) which of the userinterface elements are selectable. The location of the user object(e.g., finger) relative to the display screen can be identified in anynumber of ways including, for example, use of cameras and/or the RADARsensing system as described elsewhere herein.

In some instances, the computing device can receive a user input (e.g.,a user touch input) from the user object that selects a user interfaceelement (e.g., a user interface element that is being visuallymodified). For example, a touch or tap input can be used to select auser interface element.

In some implementations, in response to the user input, the computingdevice can provide a second user interface for presentation on thedisplay. For example the second user interface can enable the user tosearch, store for later retrieval, or communicate information associatedwith the selected user interface element.

Thus, aspects of the present disclosure provide streamlined search,storage, and retrieval of information associated with various entities.In particular, when combined with the artificial intelligence layerdescribed herein, the present disclosure provides a holistic system thatautomatically and intelligently identifies content within a userinterface that corresponds to certain entities and, using the visualmodification techniques described herein, actively alerts the userregarding the ability to select a user interface element correspondingto such content/entities to perform advanced information handlingprocedures.

The systems and methods of the present disclosure provide a number oftechnical effects and benefits. As one example, technical effect andbenefit, by streamlining the information storage and retrieval process,the number of redundant requests, queries, searches, or other actionsacross multiple different applications can be reduced, thereby providingsavings of computational and network resources. In particular, tocontinue the example above with respect to the style magazine web page,if the user is interested in shopping for the watch, the user candiscover the user interface element that corresponds to the watch (e.g.,as a result of visual modification of such interface element) and, witha small number of inputs, receive additional information about thewatch. This is in contrast to previous systems in which the user wouldhave been required to perform a web search for the watch, which mayinclude several different iterations of search queries and results dueto the inexact nature of web searching and/or the user's lack ofexplicit knowledge of the watch product name. These redundant and/orun-fruitful searches increase web traffic and consume both device andserver resources. By providing streamlined and discoverable informationretrieval systems, the systems and methods of the present disclosureeliminate such redundant and/or un-fruitful searches and, therefore,conserve processing, memory, and network bandwidth resources.

With reference now to the Figures, example embodiments of the presentdisclosure will be discussed in further detail.

Example Devices and Systems

FIG. 1 depicts a block diagram an example computing system according toexample embodiments of the present disclosure. The system includes acomputing device 102 that performs aspects of the present disclosure.The computing device 102 can be communicatively coupled to a servercomputing device 150 over a network 130.

The computing device 102 can be any form of device, such as mobilecomputing device (e.g., a smartphone, tablet, computing device that isable to be worn (e.g., computing device embedded in a pair ofeyeglasses, a wristband, a necklace, etc.), handheld computing device,etc.), computing device embedded in a vehicle, home appliance, or othersystem, smart speaker, laptop, desktop, gaming console, etc.

The computing device 102 includes one or more processors 104 and amemory 106. The one or more processors 104 can be any form of processingdevice, including, for example, a processing unit, a microprocessor, acontroller, a microcontroller, an application specific integratedcircuit, etc. The memory 106 can include one or more of anynon-transitory computer-readable medium, including, for example, RAM(e.g., DRAM), ROM (e.g., EEPROM), optical storage, magnetic storage,flash storage, solid-state storage, hard drives, or some combinationthereof. The memory 106 can store one or more sets of instructions 108that, when executed by the computing device 102, cause the computingdevice 102 to perform operations consistent with the present disclosure.

The computing device 102 can further include a network interface 118 anda display 110. The network interface 118 can enable communications overthe network 130. The network interface 118 can include any number ofcomponents to provide networked communications (e.g., transceivers,antennas, controllers, cards, etc.).

The display 110 can include different types of display components, suchas, for example, a light-emitting diode display (e.g., organiclight-emitting diode display), a liquid-crystal display (e.g.,thin-film-transistor liquid-crystal display), a thin-film diode display,etc. In some implementations, the display 110 can also betouch-sensitive. For example, the display 110 can be a capacitivetouchscreen, a resistive touchscreen, or other touch-sensitivetechnologies. Thus, in some implementations, the display 110 can serveas or be the touch-sensitive component 112. In other implementations,the computing device 102 can have an additional or alternativetouch-sensitive component 112 that is separate and distinct from thedisplay 110. For example, the touch-sensitive component 112 can be atouch pad or the like.

In some implementations, the computing device 102 can also include agyroscope 114. The gyroscope 114 can provide data that indicates theorientation of the device 102 relative to the direction of gravity(e.g., towards the ground). The gyroscope 114 can include one or moremoving (e.g., spinning) components or can include only staticcomponents. As one example, the gyroscope 114 can include an arrangementof accelerometers (e.g., three or more accelerometers arranged to detectacceleration in three dimensions).

In some implementations, the computing device 102 can also include aradio detection and ranging (RADAR) sensing system 116. The computingdevice 102 can use the RADAR sensing system 116 to determine an objectlocation of the user object (e.g., finger or stylus). For example, theRADAR sensing system 116 can emit and receive radio waves to determinethe range, angle, or velocity of objects near the computing device 102.

In some implementations, the RADAR sensing system 116 can work byemitting electromagnetic waves in a broad beam. Objects within the beamscatter this energy, reflecting some portion back towards the RADARantenna. Properties of the reflected signal, such as energy, time delay,and frequency shift capture rich information about the object'scharacteristics and dynamics, including size, shape, orientation,material, distance, and velocity.

In some implementations, the RADAR sensing system 116 can track andrecognize dynamic gestures expressed by fine motions of the fingers andhand. In order to accomplish this with a single chip sensor, in someimplementations, unlike traditional radar sensors, the RADAR sensingsystem 116 does not require large bandwidth and high spatial resolution.In particular, in some implementations, the fundamental sensingprinciples of the RADAR sensing system 116 can rely on motion resolutionby extracting subtle changes in the received signal over time. Byprocessing these temporal signal variations, the RADAR sensing system116 can distinguish complex finger movements and deforming hand shapeswithin its field.

In some implementations, the RADAR sensing system 116 can include asensor and antenna array in an ultra-compact 8 mm×10 mm package and canoperate in the 60-GHz ISM band. Two example modulation architecturesthat can be used are a Frequency Modulated Continuous Wave (FMCW) RADARand a Direct-Sequence Spread Spectrum (DSSS) RADAR. Both of these canintegrate the entire RADAR system into the package, including multiplebeamforming antennas that enable 3D tracking and imaging with no movingparts. In other implementations, the RADAR sensing system 116 can havedifferent designs and/or operating principles to those examplesdescribed above.

In some implementations, the computing device 102 can include one ormore cameras 120. For example, the cameras 120 can include front-facingcameras, rear-facing cameras, periscopic cameras, and/or configurations.The cameras 120 can include visible spectrum cameras, infrared cameras,or other forms of devices that capture imagery. The computing device 102can perform various image processing and/or computer vision techniquesto understand or process the content of the captured imagery.

In some implementations, the computing device 102 can include a userinterface manager 122 that controls or manages presentation of a userinterface on the display 110, including, for example, tasks likepresenting the user interface, modifying the user interface, receivinguser input directed to the user interface, etc. In some implementations,the user interface manager 122 can be included in an application thatgenerates the user interface. In other implementations, the userinterface manager can be separate from whichever component (e.g.,application) generated the user interface. In some implementations, theuser interface manager 122 can be an aspect or component of an operatingsystem of the computing device 102.

According to an aspect of the present disclosure, in someimplementations, the user interface manager 122 can continuously orperiodically analyze the user interface that is currently or about to bedisplayed by the computing device 102 to identify user interfaceelements that are or should be selectable by the user of the computingdevice 102. For example, the user interface manager 122 can perform someor all of the steps of methods 200 and 500 described with reference toFIGS. 2 and 5 below.

Furthermore, in some implementations, the computing device 102 caninclude an artificial intelligence layer 124. The artificialintelligence layer 124 can recognize entities within the user interface.More particularly, in some implementations, the artificial intelligencelayer 124 can automatically and intelligently identify user interfaceelements (e.g., certain items of content) within the user interface thatcorrespond to certain entities. As examples, entities can includelocations, points of interest, persons, objects, products (e.g.,articles of clothing, food items, etc.), brands, programming content(e.g., movies), bands, events, and/or other people, places, or things.

In some implementations, the artificial intelligence layer 124 canperform mobile vision techniques to recognize entities in imageryincluded in the user interface and/or text recognition techniques torecognize entities in text included in the user interface. In someimplementations, the artificial intelligence layer 124 can include oneor more machine-learned models. As examples, the machine-learned modelscan include machine-learned image recognition models, machine-learnedobject detection or recognition models, machine-learned facial detectionor recognition models, machine-learned text recognition models, and/orvarious other types of machine-learned models that are useful forrecognizing and understanding entities included in user interfacecontent. These machine-learned models can, in some implementations,include artificial neural networks. Example artificial neural networksinclude feed-forward neural networks, recurrent neural networks, andconvolutional neural networks. Neural networks can be deep neuralnetworks.

In some implementations, the artificial intelligence layer 124 can beincluded in an application that generates the user interface. In otherimplementations, the artificial intelligence layer 124 can be separatefrom whichever component (e.g., application) generated the userinterface. In some implementations, the artificial intelligence layer124 can be an aspect or component of an operating system of thecomputing device 102.

Each of the user interface manager 122 and the artificial intelligencelayer 124 can include computer logic utilized to provide desiredfunctionality. Each of the user interface manager 122 and the artificialintelligence layer 124 can be implemented in hardware, firmware, and/orsoftware controlling a general purpose processor. For example, in someimplementations, each of the user interface manager 122 and theartificial intelligence layer 124 includes program files stored on astorage device, loaded into a memory and executed by one or moreprocessors. In other implementations, each of the user interface manager122 and the artificial intelligence layer 124 includes one or more setsof computer-executable instructions that are stored in a tangiblecomputer-readable storage medium such as RAM hard disk or optical ormagnetic media.

In some implementations, the computing device 102 can communicativelyconnect to a server computing device 150 over the network 130. Theserver computing device 150 can include one or more processors 152 and amemory 154. The one or more processors 152 can be any form of processingdevice, including, for example, a processing unit, a microprocessor, acontroller, a microcontroller, an application specific integratedcircuit, etc. The memory 154 can include one or more of anynon-transitory computer-readable medium, including, for example, RAM(e.g., DRAM), ROM (e.g., EEPROM), optical storage, magnetic storage,flash storage, solid-state storage, hard drives, or some combinationthereof. The memory 154 can store one or more sets of instructions 156that, when executed by the server computing device 150, cause the servercomputing device 150 to perform operations consistent with the presentdisclosure.

The server computing device 150 can further include a network interface159. The network interface 159 can enable communications over thenetwork 130. The network interface 159 can include any number ofcomponents to provide networked communications (e.g., transceivers,antennas, controllers, cards, etc.).

The network 130 can be any type of communications network, such as alocal area network (e.g., intranet), wide area network (e.g., Internet),or some combination thereof and can include any number of wired orwireless links. In general, communication between the server computingdevice 150 and the computing device 102 can be carried via any type ofwired and/or wireless connection, using a wide variety of communicationprotocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g.,HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).Server computing device 150 can communicate with the computing device102 over network 130 by sending and receiving data.

Further, any of the processes, operations, programs, applications, orinstructions described as being stored at or performed by the servercomputing device 150 can instead be stored at or performed by thecomputing device 102 in whole or in part, and vice versa.

Example Methods

FIG. 2 depicts a flow chart diagram of an example method 200 to increasediscoverability of selectable user interface elements according toexample embodiments of the present disclosure.

At 202, a computing device can provide a user interface that includes aplurality of user interface elements for presentation on a display. Forexample, the user interface can be a user interface generated by anapplication of the computing device. As examples, user interfaceelements can include icons, buttons, textual elements, items of content,pictures (e.g., pictures being captured in real-time via a camera and/orpreviously captured pictures), banners, titles, headers, entry fields,windows, menus, controls, graphics, and/or the like.

As one example user interface, a web browser application can provide auser interface that includes a web page that includes a number of userinterface elements. As another example, a text messaging application canprovide a user interface that includes text messages sent and/orreceived to one or more contacts. As yet another example, a cameraapplication can provide a user interface that depicts a field of view ofa camera of the computing device. Many other and different userinterfaces can be provided as well.

At 204, the computing device can identify one or more of the userinterface elements that are selectable by a user of the computingdevice. For example, in some implementations, the computing device caninclude a user interface manager. In some implementations, the userinterface manager can be separate from whichever component (e.g.,application) generated the user interface. The user interface managercan continuously or periodically analyze the user interface that iscurrently or about to be displayed by the computing device to identifyuser interface elements that are or should be selectable by the user ofthe computing device.

According to an aspect of the present disclosure, in someimplementations, identifying the one or more of the user interfaceelements that are selectable at 204 can include employing an artificialintelligence layer to recognize entities within the user interface. Moreparticularly, in some implementations, a computing device can include anartificial intelligence layer that automatically and intelligentlyidentifies user interface elements (e.g., certain items of content)within a user interface that correspond to certain entities. Asexamples, entities can include locations, points of interest, persons,objects, products (e.g., articles of clothing, food items, etc.),brands, programming content (e.g., movies), bands, events, and/or otherpeople, places, or things.

In some implementations, the artificial intelligence layer can performmobile vision techniques at 204 to recognize entities in imageryincluded in the user interface and/or text recognition techniques torecognize entities in text included in the user interface. In someimplementations, the artificial intelligence layer can include one ormore machine-learned models. As examples, the machine-learned models caninclude machine-learned image recognition models, machine-learned objectdetection or recognition models, machine-learned facial detection orrecognition models, machine-learned text recognition models, and/orvarious other types of machine-learned models that are useful forrecognizing and understanding entities included in user interfacecontent.

In some implementations, identifying user interface elements that areselectable at 204 can include generating additional user interfaceelements for presentation within the user interface and/or transforminga certain user interface element into a multiple user interfaceelements. Thus, new and/or modified user interface elements can be usedto enable the user to select certain specific entities included in alarger group of entities.

In some implementations, identifying user interface elements that areselectable at 204 can include accessing attributes or parametersassociated with the user interface elements. For example, in someinstances, the component (e.g., application) that generated the userinterface can provide attribute or parameter data for each userinterface element included in the user interface, including, forexample, data that indicates whether each user interface element isselectable. Thus, the user interface manager can access or read thisdata to identify which of the user interface elements are selectable.

At 206, the computing device can visually modify at least a first userinterface element of the one or more selectable user interface elements.As an example, in some implementations, all of the user interfaceelements that are selectable by the user can be visually modified at206. In other implementations, only a subset of the selectable userinterface elements can be visually modified at 206 (e.g., based onvarious contextual data as described elsewhere herein).

In some implementations, at 206, the computing device can visuallymodify the one or more user interface elements as soon as elements loadwithin the user interface (e.g., as soon as the application, web page,etc. being displayed loads). In other implementations, the computingdevice can wait to visually modify the one or more user interfaceelements until some other event occurs such as, for example, when a userinvokes the keyboard, when the user performs a scrolling action, whenthe user reaches an end of the current interface or page, and/or otheractions or events.

One example visual modification that the computing device can perform at206 is to visually modify a user interface element to have a gleamappearance in which the user interface element appears to reflect light.For example, the gleam appearance can visually mimic the appearance ofsun reflecting off of a watch face, a body of water, or other reflectiveitem. Visually modifying a user interface element to have a gleamappearance can bring the user's attention to the user interface elementand the fact that it is selectable.

Another example visual modification includes outlining the userinterface element (e.g., using a bolder or darker outline or shadowboxthan previously used). Another example visual modification includeschanging the color of the user interface element. Another example visualmodification includes highlighting the user interface element bybrightening the user interface element and/or an area around the userinterface element. For example, a flashing of a highlighting color,shading, and/or texture on and/or around the user interface element canindicate that the user interface element is selectable.

According to another aspect of the present disclosure, in someimplementations, the visual modification performed at 206 can be basedon an orientation of the device. For example, the visual modificationcan change in response to a change in the orientation of the device.

As one example, visually modifying a user interface element based on theorientation of the device can include, for a user interface element witha gleam appearance, changing a reflection location at which the userinterface element appears to reflect light based at least in part on theorientation of the device. Thus, in one example, as the orientation ofthe device changes the reflection location can be updated, furthergiving the appearance that the user interface element is gleaming withlight.

As another example, visually modifying the user interface element basedon the orientation can include visually modifying the user interfaceelement at certain orientations but not at other orientations. Thus, inone example, if the user tilts the device to the right or left, the userinterface object can be visually modified, while if the user holds thedevice in a neutral orientation, the user interface object is notvisually modified.

In some implementations, the orientation of the device can be anorientation relative to a direction of gravity (e.g., the directiontowards the ground). As one example, FIGS. 3A and 3B depict an examplemodification of an example user interface element based on anorientation of an example computing device relative to the direction ofgravity according to example embodiments of the present disclosure.

In particular, in FIG. 3A, a mobile computing device 302 has a firstorientation (e.g., tilted slightly to the left). The computing device302 is displaying a user interface 304. The user interface 304 includesa number of different user interface elements. As an example, the userinterface 304 includes a user interface element 306. The user interfaceelement 306 is a picture of a food dish. The user interface element 306is selectable by the user (e.g., to receive additional information aboutthe food dish).

To alert the user to the fact that the user interface element 306 isselectable, the computing device 302 can visually modify the userinterface element 306. In particular, the computing device 302 canmodify the element 306 to have a gleam appearance in which the userinterface element 306 appears to reflect light.

In particular, the computing device 302 can determine a reflectionlocation 308 based on the orientation of the device 302. The reflectionlocation 308 can be the location at which the element 306 is modified toappear as most reflective of light. Thus, for example, the element 306can be modified to have a gleam or glare at the reflection location 308.

Some or all of the element 306 can be modified. For example, asillustrated in FIG. 3A, only a portion has been visually modified tohave a gleam. However, in other instances, the entirety of the element306 can be modified to have the gleam. For example, the entirety of theelement 306 can gleam, but the gleam can be brightest at the reflectionlocation 308.

According to an aspect of the present disclosure, the visualmodification of the element 306 can change over time as the orientationof the device 302 changes. In particular, in FIG. 3B, the orientation ofthe device 302 has changed so that the device is tilting slightly to theright. As a result, the visual modification can change. In particular,the computing device 302 can change the reflection location of the gleamappearance as the orientation of the device 302 changes. For example, atthe device orientation of FIG. 3B, the reflection location has moved toa new location 310.

As one example, the reflection location can slide from location 308 inFIG. 3A to location 310 in FIG. 3B as the orientation of the device 302changes from FIG. 3A to FIG. 3B. Likewise, the reflection location canslide back to location 308 if the orientation of the device 302 returnsto the orientation of FIG. 3A. Thus, the reflection location can beupdated in real time based on the orientation of the device relative tothe ground, thus causing the user interface element to gleam with light.

Referring again to FIG. 2, as another example of the modificationperformed at 206 based on the orientation of the device, the orientationof the device can be an orientation relative to a user object. The userobject can be positioned in a number of different positions relative tothe device. As one example, in some instances, the user object can hoverover the display. That is, the user object can be placed adjacent to thedisplay but not physically touching the display.

For example, the user object can be a finger or stylus associated withthe user. For example, the finger or stylus can be located at a numberof different locations relative to a display of the computing device,including, for example, hovering over the display as described above. Insome implementations, the computing device can include a RADAR sensingsystem. The computing device can use the RADAR sensing system todetermine an object location of the user object (e.g., finger orstylus). The computing device can determine the orientation of thecomputing device relative to the user object based on the determinedobject location. The visual modification of the user interface elementcan be modified (e.g., in real time) as such orientation changes.

In another example, the user object can be a face of the user. In someimplementations, the computing device can include one or more cameras.The computing device can determine a face location of the user's facebased on imagery captured by the one or more cameras. The computingdevice can determine the orientation of the computing device relative tothe user's face based on the determined face location. The visualmodification of the user interface element can be modified (e.g., inreal time) as such orientation changes.

FIGS. 4A and 4B depict an example modification of an example userinterface element based on an orientation of an example computing devicerelative to an example user object according to example embodiments ofthe present disclosure.

In particular, in FIG. 4A, a mobile computing device 402 has a firstorientation relative to the location of a user's finger 408. Thecomputing device 402 is displaying a user interface 404. The userinterface 404 includes a number of different user interface elements. Asan example, the user interface 404 includes a user interface element406. The user interface element 406 is a picture of a food dish. Theuser interface element 406 is selectable by the user (e.g., to receiveadditional information about the food dish).

To alert the user to the fact that the user interface element 406 isselectable, the computing device 402 can visually modify the userinterface element 406. In particular, the computing device 402 canmodify the element 406 to have a gleam appearance in which the userinterface element 406 appears to reflect light.

In particular, the computing device 402 can determine a reflectionlocation 410 based on the orientation of the device 402 relative to thefinger 408. The reflection location 410 can be the location at which theelement 406 is modified to appear as most reflective of light. Thus, forexample, the element 406 can be modified to have a gleam or glare at thereflection location 410.

Some or all of the element 406 can be modified. For example, asillustrated in FIG. 4A, only a portion has been visually modified tohave a gleam. However, in other instances, the entirety of the element406 can be modified to have the gleam. For example, the entirety of theelement 406 can gleam, but the gleam can be brightest at the reflectionlocation 410.

According to an aspect of the present disclosure, the visualmodification of the element 406 can change over time as the orientationof the device 402 relative to the finger 408 changes. In particular, inFIG. 4B, the location of the finger 408 has changed, thereby resultingin a change in the orientation of the device 402 relative to the finger408. As a result, the visual modification can change. In particular, thecomputing device 402 can change the reflection location of the gleamappearance as the orientation of the device 402 changes. For example, atthe device orientation of FIG. 4B, the reflection location has moved toa new location 412.

As one example, the reflection location can slide from location 410 inFIG. 4A to location 412 in FIG. 4B as the orientation of the device 402to the finger 408 changes from FIG. 4A to FIG. 4B. Likewise, thereflection location can slide back to location 410 if the orientation ofthe device 402 returns to the orientation of FIG. 4A. Thus, thereflection location can be updated in real time based on the orientationof the device relative to the finger 408, thus causing the userinterface element 406 to gleam with light.

Referring again to FIG. 2, in some implementations, the visualmodification performed at 206 can only be temporary in nature. In oneexample, as soon as a user interface (e.g., web page) loads, thecomputing device can briefly (e.g., for one half second) visually modifythe selectable user interface elements (e.g., recognized products) andthen return the elements to a neural or normal state. This can provide aquick indication to the user of which elements are selectable withoutbeing overly disruptive of the user experience.

In some implementations, the particular type of visual modification usedfor a user interface element at 206 can be based on or a function of acontent type associated with such user interface element. For example,textual elements can be modified in a first way while pictorial orgraphical elements can be modified in a second, different way.

FIG. 5 depicts a flow chart diagram of an example method 500 to increasediscoverability of selectable user interface elements according toexample embodiments of the present disclosure.

At 502, a computing device can provide a user interface that includes aplurality of user interface elements for presentation on a display. Forexample, the user interface can be a user interface generated by anapplication of the computing device. As examples, user interfaceelements can include icons, buttons, textual elements, items of content,pictures (e.g., pictures being captured in real-time via a camera and/orpreviously captured pictures), banners, titles, headers, entry fields,windows, menus, controls, graphics, and/or the like.

As one example user interface, a web browser application can provide auser interface that includes a web page that includes a number of userinterface elements. As another example, a text messaging application canprovide a user interface that includes text messages sent and/orreceived to one or more contacts. As yet another example, a cameraapplication can provide a user interface that depicts a field of view ofa camera of the computing device. Many other and different userinterfaces can be provided as well.

At 504, the computing device can identify one or more of the userinterface elements that are selectable by a user of the computingdevice. For example, in some implementations, the computing device caninclude a user interface manager. In some implementations, the userinterface manager can be separate from whichever component (e.g.,application) generated the user interface. The user interface managercan continuously or periodically analyze the user interface that iscurrently or about to be displayed by the computing device to identifyuser interface elements that are or should be selectable by the user ofthe computing device.

According to an aspect of the present disclosure, in someimplementations, identifying the one or more of the user interfaceelements that are selectable at 504 can include employing an artificialintelligence layer to recognize entities within the user interface. Moreparticularly, in some implementations, a computing device can include anartificial intelligence layer that automatically and intelligentlyidentifies user interface elements (e.g., certain items of content)within a user interface that correspond to certain entities. Asexamples, entities can include locations, points of interest, persons,objects, products (e.g., articles of clothing, food items, etc.),brands, programming content (e.g., movies), bands, events, and/or otherpeople, places, or things.

In some implementations, the artificial intelligence layer can performmobile vision techniques at 504 to recognize entities in imageryincluded in the user interface and/or text recognition techniques torecognize entities in text included in the user interface. In someimplementations, the artificial intelligence layer can include one ormore machine-learned models. As examples, the machine-learned models caninclude machine-learned image recognition models, machine-learned objectdetection or recognition models, machine-learned facial detection orrecognition models, machine-learned text recognition models, and/orvarious other types of machine-learned models that are useful forrecognizing and understanding entities included in user interfacecontent.

In some implementations, identifying user interface elements that areselectable at 504 can include generating additional user interfaceelements for presentation within the user interface and/or transforminga certain user interface element into a multiple user interfaceelements. Thus, new and/or modified user interface elements can be usedto enable the user to select certain specific entities included in alarger group of entities.

In some implementations, identifying user interface elements that areselectable at 504 can include accessing attributes or parametersassociated with the user interface elements. For example, in someinstances, the component (e.g., application) that generated the userinterface can provide attribute or parameter data for each userinterface element included in the user interface, including, forexample, data that indicates whether each user interface element isselectable. Thus, the user interface manager can access or read thisdata to identify which of the user interface elements are selectable.

At 506, the computing device can determine an object location of a userobject relative to the display. The object location can be a location inthree-dimensional space. Alternatively, the object location can be alocation in two-dimensional space relative to the X,Y coordinates of thedisplay screen (e.g., the X,Y coordinates of the display screen thatmost closely match the object location.

As examples, the user object can be a finger of the user, a styluscontrolled by the user, a face of the user, and/or other user objects.For example, the user object can be located adjacent to the display butnot physically touching the display (i.e., hovering over the display)

In one example, determining the object location at 506 can include usinga RADAR sensing system to determine the object location. In anotherexample, determining the object location at 506 can include analyzingimagery captured by one or more cameras of the computing device todetermine the object location.

At 508, the computing device can determine whether the object locationcorresponds to one of the selectable user interface elements. Forexample, in some implementations, the object location can be projectedonto the display screen to determine whether the object locationcorresponds to one of the selectable user interface elements (e.g.,whether the projected object location overlaps a boundary areaassociated with one of the elements).

If it is determined at 508 that the object location does not correspondto one of the selectable user interface elements, then method 500 canreturn to 506 and again determine the object location. However, if it isdetermined at 508 that the object location does not correspond to one ofthe selectable user interface elements, then method 500 can proceed to510.

At 510, the computing device can visually modify the user interfaceelement to which the object location corresponds. Any of the differentvisual modification described herein can be performed at 510.

In one example, a user may use her finger to hover over the display ofthe device. When the user's finger is located over (or within a certaindistance of) a user interface element that is selectable, the computingdevice can visually modify such user interface element (e.g., using anycombination of one or more of the visual modifications describedherein).

At 514, the computing device can determine whether the user selected theuser interface element that was visually modified at 510. If it isdetermined at 514 that the user did not select the user interfaceelement, then method 500 can return to 506 and again determine theobject location. However, if it is determined at 514 that the userselected the user interface element, then method 500 can proceed to 516.

At 516, the computing device can provide a new user interface thatenables the user to search, store, and/or communicate informationassociated with the user interface element.

FIGS. 6A-D depict an example modification of example user interfaceelements based on a location of an example user object relative to theuser interface elements according to example embodiments of the presentdisclosure.

Referring first to FIG. 6A, a computing device 602 is displaying a userinterface 604. The user interface 604 includes a number of userinterface elements, including, for example, user interface elements 606,608, and 610. A user is holding her finger at an object location 612.The object location 612 does not correspond to any of the user interfaceelements.

Referring now to FIG. 6B, the user has moved her finger to change theobject location 612. In FIG. 6B, the object location 612 does correspondto the user interface element 610. In particular, as an example, aprojection of the object location 612 onto the user interface 604 iswithin an area associated with the user interface element 610. As aresult, the computing device 602 can visually modify the user interfaceelement 610. For example, as illustrated in FIG. 6B, element 610 hasbeen visually modified to have a bolder outline.

Referring now to FIG. 6C, the user has again moved her finger to changethe object location 612. In FIG. 6C, the object location 612 correspondsto the user interface element 608. In particular, as an example, aprojection of the object location 608 onto the user interface 604 iswithin an area associated with the user interface element 608. As aresult, the computing device 602 can visually modify the user interfaceelement 608. For example, as illustrated in FIG. 6C, element 608 hasbeen visually modified to have a bolder outline. Notably, element 610does not have the bolder outline any longer.

Referring now to FIG. 6D, the user has again moved her finger to pressthe display at the location of user interface element 608, therebyselecting user interface element 608. As a result, the computing devicehas provided an additional or modified user interface 614 on thedisplay. The additional user interface 614 can enable the user tosearch, store, and/or share information about the user interface element608 (e.g., about its associated entity such as a food dish “paella”).

Thus, a user may hover her finger over the display screen and move herfinger to different locations to discover (e.g., via the resultingvisual modification of the user interface elements) which of the userinterface elements are selectable. The location of the user object(e.g., finger) relative to the display screen can be identified in anynumber of ways including, for example, use of cameras, use of capacitivesensing technologies, and/or the RADAR sensing system as describedelsewhere herein.

ADDITIONAL DISCLOSURE

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

In particular, although FIGS. 2 and 5 respectively depict stepsperformed in a particular order for purposes of illustration anddiscussion, the methods of the present disclosure are not limited to theparticularly illustrated order or arrangement. The various steps of themethods 200 and 500 can be omitted, rearranged, combined, and/or adaptedin various ways without deviating from the scope of the presentdisclosure.

What is claimed is:
 1. A computing device, comprising: one or moreprocessors; a display; and one or more non-transitory computer-readablemedia that collectively store instructions that, when executed by theone or more processors, cause the computing device to performoperations, the operations comprising: providing a user interface thatcomprises a plurality of user interface elements for presentation on thedisplay; identifying one or more of the user interface elements that areselectable by a user of the computing device; and visually modifying atleast a first user interface element of the one or more selectable userinterface elements to have a gleam appearance in which the first userinterface element appears to reflect light.
 2. The computing device ofclaim 1, wherein the operations further comprise: determining anorientation of the computing device relative to a user object associatedwith a user of the computing device; wherein the operation of visuallymodifying the first user interface element to have the gleam appearancecomprises: determining a reflection location within the first userinterface element based at least in part on the orientation of thecomputing device relative to the user object; and visually modifying thefirst user interface element to have the gleam appearance in which thefirst user interface element appears to reflect light at the reflectionlocation within the first user interface element.
 3. The computingdevice of claim 2, wherein the user object comprises a finger or astylus located adjacent to the display but not physically touching thedisplay.
 4. The computing device of claim 2, wherein the computingdevice further comprises: a RADAR sensing system that senses an objectlocation of the user object; wherein the operation of determining theorientation of the computing device relative to the user objectcomprises determining the orientation of the computing device relativeto the object location sensed by the RADAR sensing system.
 5. Thecomputing device of claim 2 wherein: the user object comprises a face ofthe user; the computing device further comprises one or more camerasthat capture imagery; the operations further comprise determining a facelocation of the face of the user based on the imagery captured by theone or more cameras; and the operation of determining the orientation ofthe computing device relative to the user object comprises determiningthe orientation of the computing device relative to the face location ofthe face of the user.
 6. The computing device of claim 1, wherein: thecomputing device further comprises a gyroscope; the operations furthercomprise determining an orientation of the computing device based ondata produced by the gyroscope; and the operation of visually modifyingthe first user interface element to have the gleam appearance comprises:determining a reflection location within the first user interfaceelement based at least in part on the orientation of the computingdevice; and visually modifying the first user interface element to havethe gleam appearance in which the first user interface element appearsat the reflection location.
 7. The computing device of claim 1, whereinthe operation of identifying the one or more of the user interfaceelements that are selectable by the user of the computing devicecomprises using one or more machine-learned models to recognize entitieswithin the user interface.
 8. The computing device of claim 3, furthercomprising, after visually modifying the first user interface element:receiving a user touch input from the user object that selects the firstuser interface element; and in response to the user touch input,providing a second user interface for presentation on the display,wherein the second user interface enables the user to one or more ofsearch, store, or communicate information associated with the first userinterface element.
 9. A computer-implemented method, comprising:providing, by one or more computing devices, a user interface thatcomprises a plurality of user interface elements for presentation on adisplay; identifying, by the one or more computing devices, one or moreof the user interface elements that are selectable by a user;determining, by the one or more computing devices, an object location ofa user object relative to the display, wherein the user object comprisesa finger or a stylus located adjacent to the display but not physicallytouching the display; determining, by the one or more computing devices,whether the object location of the user object corresponds to one of theone or more user interface elements that are selectable by the user; andwhen the object location corresponds to a first user interface elementof the one or more selectable user interface elements, visuallymodifying, by the one or more computing devices, the first userinterface element to visually highlight the first user interfaceelement.
 10. The computer-implemented method of claim 9, whereindetermining, by the one or more computing devices, whether the objectlocation of the user object corresponds to one of the one or more userinterface elements that are selectable by the user comprisesdetermining, by the one or more computing devices, whether the finger orthe stylus is hovering over one of the one or more user interfaceelements that are selectable by the user.
 11. The computer-implementedmethod of claim 9, wherein visually modifying, by the one or morecomputing devices, the first user interface element to visuallyhighlight the first user interface element comprises temporarilyvisually modifying, by the one or more computing devices, the first userinterface element to visually highlight the first user interfaceelement.
 12. The computer-implemented method of claim 9, whereinvisually modifying, by the one or more computing devices, the first userinterface element to visually highlight the first user interface elementcomprises visually modifying, by the one or more computing devices, thefirst user interface element according to an orientation of the displayrelative to the object location of the user object.
 13. Thecomputer-implemented method of claim 12, wherein visually modifying, bythe one or more computing devices, the first user interface elementaccording to the orientation of the display relative to the objectlocation of the user object comprises: visually modifying, by the one ormore computing devices, the first user interface element to have a gleamappearance in which the first user interface element appears to reflectlight; wherein a reflection location at which the first user interfaceelement appears to reflect light is based at least in part on theorientation of the display relative to the object location of the userobject.
 14. The computer-implemented method of claim 9, wherein visuallymodifying, by the one or more computing devices, the first userinterface element to visually highlight the first user interface elementcomprises: determining, by the one or more computing devices, a firstcontent type associated with the first user interface element; andvisually modifying, by the one or more computing devices, the first userinterface element according to a first modification type selected from aplurality of different available modification types that arerespectively associated with a plurality of different content types. 15.The computer-implemented method of claim 9, wherein determining, by theone or more computing devices, the object location of the user objectrelative to the display comprises receiving, by the one or morecomputing devices, RADAR data from a RADAR sensing system that describesthe object location of the user object relative to the display.
 16. Thecomputer-implemented method of claim 9, wherein identifying, by the oneor more computing devices, the one or more of the user interfaceelements that are selectable by the user comprises using, by the one ormore computing devices, one or more machine-learned models to recognizeentities within the user interface.
 17. The computer-implemented methodof claim 9, further comprising: receiving, by the one or more computingdevices, a user touch input from the user object that selects the firstuser interface element; and in response to the user touch input,providing, by the one or more computing devices, a second user interfacefor presentation on the display, wherein the second user interfaceenables the user to store information associated with the first userinterface element for later retrieval.
 18. Apparatus configured: toprovide a user interface that comprises a plurality of user interfaceelements for presentation on a display; to identify one or more of theuser interface elements that are selectable by a user; and to modifydisplay of selectable user interface elements as a function of alocation of a finger or stylus located adjacent to the display but notphysically touching the display.
 19. Apparatus as claimed in claim 18,wherein the apparatus is configured to display the one or moreselectable user interface elements without modification in the presenceof a first condition and to modify in real time display of selectableuser interface elements as a function of the location of the finger orstylus located adjacent to the display but not physically touching thedisplay in the absence of the first condition, the first condition beinga neutral orientation of the apparatus.
 20. Apparatus as claimed inclaim 18, wherein the apparatus comprises a RADAR sensing systemconfigured to sense the location of the finger or stylus locatedadjacent to the display but not physically touching the display.